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1.  The  purpose  of  this  letter  is  to  announce  the  issuance  of  Appendix  B  to  FPM  Supplement 
335-1.   (Supplement  335-1,  "Evaluation  of  Employees  for  Promotion  and  Internal  Placement,' 
was  issued  through  FPM  Letter  335-12  dated  December  29,  1978.) 

2.  A  draft  of  Appendix  B,  "Guidelines  for  Evaluating  Employees  for  Promotion  and  Internal 
placement,"  was  transmitted  to  agencies  and  unions  for  comment.   Only  ten  comments 
were  received,  indicating  basic  concurrence  with  the  document.   Many  of  the  comments, 
however,  did  contain  suggestions  for  specific  changes.   The  final  version  reflects 
revisions  made  as  a  result  of  these  suggestions. 

3.  Appendix  B  contains  useful  advisory  material  in  this  area  but  is  not  binding  on 
agencies. 


4.   This  Appendix  reviews  certain  concepts  and  problems  in  evaluating  employees  for  promo- 
tion and  internal  placement  and  describes  methods  and  necessary  resources  for 
agencies  to  carry  out  the  principles  set  forth  in  the  supplement.   Topics  discussed 
in  the  appendix  include,  for  example,  needed  technical  competence,  description  of 
the  evaluative  process,  types  of  evaluation  methods,  determining  effectiveness  of 
instruments,  and  information  on  widely  used  OPM  evaluation  procedures. 
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(e)  Ability  to  plan  objectives  and  staffing  and  resource  requirements  to  implement 
the  personnel  measurement  program  throughout  the  agency. 

(f)  Ability  to  determine  training  needs  in  personnel  measurement  areas  and  to  carry 
out  training  activities  designed  to  meet  objectives  of  the  personnel  measurement  program. 

c.  Personnel  measurement  capability  needed  by  personnel  specialists  and  technicians. 

(1)  Many  recurring  staffing  activities  which  involve  personnel  measurement  (e.g. 
determination  and  documentation  of  work  behaviors,  skills,  knowledges,  and  abilities; 
development  of  vacancy  announcements;  development  or  modification  of  instruments  for 
performance  appraisal;  evaluation  of  training,  education  and  experience;  administration 
of  tests  and  other  procedures;  monitoring  of  rating  panels)  are  carried  out  by  personnel 
specialists.   Program  effectiveness  is  largely  dependent  upon  the  extent  to  which  the 
personnel  specialists  are  able  to  apply  appropriate  job  analysis  procedures  and  psycho- 
metric principles  in  these  activities.   Special  attention  to  selection  for  these  posi- 
tions and  to  training  programs  for  current  staff  may  be  warranted. 

(2)  Major  attention  should  be  given  to  structuring  the  work  activities  in  personnel 
measurement  areas  to  best  utilize  staff  resources.   Some  recurring  data  operations  and 
statistical  procedures,  for  example,  may  best  be  carried  out  by  personnel  technicians 
or  psychology  technicians  under  the  guidance  of  staffing  specialists  of  personnel 
psychologists. 

(3)  Employee  development  specialists  and  EEO  specialists  in  many  instances  need 
specialized  abilities  in  personnel  measurement.   A  special  awareness  is  also  needed 
in  these  areas  of  the  relationship  and  distinctions  among  objectives  and  activities 
of  counseling  and  employee  development,  equal  employment  opportunity  and  affirmative 
action,  and  merit  promotion  and  placement. 

d.  Personnel  measurement  capabilities  needed  by  supervisors  and  selecting  officials. 

(1)  Supervisors,  as  well  as  other  employees,  are  frequently  primary  information 
sources  for  information  concerning  jobs  to  be  filled  and  the  important  requirements 
for  successful  performance  on  the  job.   They  need  to  understand  and  support  the  job 
analysis  and  measurement  activities  of  the  personnel  office.   Orientation  sessions  and 
materials  for  supervisory  personnel  on  objectives  and  planned  procedures  for  improving 
personnel  measurement  can  facilitate  their  support.   As  improved  personnel  measurement 
procedures  yield  results  in  terms  of  placing  the  best  qualified  persons  on  the  job, 
the  support  of  managers  and  supervisors  will  be  further  strengthened. 

(2)  Supervisory  appraisals  of  employee  performance  are  usually  of  major  importance 

in  personnel  measurement  programs,  both  as  selection  procedures  themselves  and  as  criteria 
against  which  the  predictive  validity  of  other  procedures  is  evaluated.   Substantial 
research  indicates  that  the  relevance,  objectivity,  reliability,  and  validity  of 
supervisory  appraisals  is  enhanced  when  supervisors  are  provided  special  training  on 
making  appraisals  and  how  to  avoid  common  types  of  rater  error. 

(3)  Selecting  officials  need  information  concerning  the  rating  process  to  assist 
them  in  making  decisions  from  the  list  of  best-qualified  candidates.   They  need  to 
know  whether  the  differences  in  final  rating  of  those  referred  are  likely  to  be 
important  in  terms  of  job  performance  differences.   Usually  this  can  be  best  accomplished 
through  orientation  of  selecting  officials  concerning  the  selection  instruments  and 
procedures  used  for  jobs  under  their  cognizance  and,  additionally,  through  providing 
interpretive  information  in  conjunction  with  each  certificate  which  discusses  the 
degree  of  importance  of  differences  in  candidate  ranking  on  that  particular  certificate. 

B-4.   OVERVIEW  OF  THE  EVALUATION  PROCESS 

a.   General.   Except  when  screening  criteria  have  been  used  to  retrieve  information 
from  a  skills  file,  an  initial  review  of  all  candidates  in  the  area  of  consideration 
or  all  who  respond  to  a  vacancy  announcement  will  be  necessary  to  screen  out 
unqualified  candidates.   The  following  are  two  approaches  which  may  be  used  for 
identifying  the  candidates  for  competitive  evaluation: 

(1)   Separate  screening  and  ranking.   Screen  all  candidates  on  the  basis  of  elements 
that  are  essential  to  adequate  performance  in  the  specific  position  to  be  filled.   A 
careful  and  thorough  job  analysis  is  needed  to  identify  only  those  elements  actually 


Attachment  to  FPM  Ltr.  335-  13  (3) 


essential  to  minimum  acceptable  performance.   It  is  important  that  screenout  criteria 
distinguish  only  between  those  who  could  perform  adequately  in  the  job  and  those  who 
could  not.   It  is  also  important  to  avoid  use  of  search  or  screenout  criteria  which 
unduly  restrict  competition  or  impose  artificial  barriers  to  advancement,  for  instance 
by  including  as  an  element  a  skill  which  can  be  easily  learned.   Unless  a  job  element 
qualification  standard  is  authorized  by  the  Office  of  Personnel  Management  (OPM)  for 
internal  placement  in  the  position  being  filled,  care  should  be  taken  that  persons 
screened  by  this  method  meet  the  OPM's  minimum  qualification  requirements  or  the 
requirements  under  such  agency  standards  as  may  be  authorized. 

(2)   Combining  screening  and  ranking.   When  the  number  of  candidates  is  small,  it 
may  be  possible  to  eliminate  the  separate  screening  stage  and  perform  the  basic  qualifica- 
tion determination  and  competitive  evaluation  at  the  same  time.   Those  found  unquali- 
fied for  the  job  could  be  so  notified  and  eliminated  from  further  consideration.   If 
this  procedure  is  used  by  an  evaluation  panel,  it  may  be  necessary  for  advisory  partici- 
pation by  a  personnel  office  representative  to  assure  that  the  qualification  standards 
are  understood  and  that  all  OPM  requirements  are  met. 

b.  Evaluation  procedures.   The  overall  objectives  of  evaluation  are  to  narrow  the 
number  of  eligible  candidates  to  a  reasonable  number  from  which  a  selection  may  be 
made  and  to  assure  that  selection  is  made  from  among  the  best  qualified  candidates. 
It  is  necessary  that  candidates  are  evaluated  on  the  basis  of  predetermined  criteria 
to  establish  their  relative  merit  for  promotion  to  the  position  being  filled.   To 
achieve  these  objectives,  evaluation  procedures  should: 

(1)  Be  based  on  a  thorough  analysis  of  the  job  to  determine  the  work  behaviors 
and  knowledges,  skills,  abilities  and  other  characteristics  to  be  used  to  measure 
the  probability  or  expectancy  of  successful  performance  among  candidates; 

(2)  Provide  for  an  evaluation  of  each  candidate's  qualifications  against  the  work 
behaviors  of  KSAO's  required  for  successful  performance  of  the  target  position; 

(3)  Result  in  a  distinction  among  candidates  on  the  basis  of  relative  qualification 
1  eve  1 . 

(4)  Provide  for  a  consideration  of  those  qualifications  which  indicate  a  candidate's 
potential  for  future  promotion,  when  the  job  being  filled  leads  to  further  advancement. 
(See  Section  51  of  Appendix  A). 

c.  Criteria.   Criteria,  as  used  in  the  evaluation  process,  are  the  standards  against 
which  the  qualifications  of  candidates  are  assessed.   In  order  for  each  criterion  to 
be  useful  for  rating  purposes,  it  must  be  important  to  successful  performance  of  the 
duties  of  the  position  to  be  filled;  not  be  subject  to  being  learned  or  acquired 

on  the  job  within  a  short  period  of  time  after  promotion  or  placement;  and  distin- 
quish  among  candidates  according  to  expected  levels  of  job  performance  (e.g.,  successful 
versus  superior).   Failure  to  meet  these  requirements  tends  to  result  in  placing 
artificial  limits  on  competition,  to  foster  inbreeding, and  to  restrict  the  effective- 
ness of  manpower  planning  and  career  development  by  the  agency. 

Establishment  of  specialized  evaluation  factors  which  far  exceed  actual  job  demands 
may  suggest  an  attempt  to  tailor  the  job  to  a  preselected  candidate. 

(1)  Minimum  qualification  standards,  such  as  those  in  OPM  Handbooks  X-118  and 
X-118C,  are  used  to  eliminate  those  persons  who  are  not  eligible  for  further 
consideration  because  of  the  low  expectancy  that  those  who  fail  to  meet  the  standard 
will  perform  the  duties  of  the  job  to  be  filled  at  at  least  a  minimally  acceptable 
level.   These  standards  are  set  at  a  level  which  maximize  the  probability  of  correctly 
identifying  and  di stinqui shing  between  potentially  satisfactory  and  potentially 

poor  job  performers. 

(2)  Rating  criteria  can  also  be  used  to  identify  well  qualified  candidates  when 
the  standards  are  set  at  a  level  which  maximizes  the  probability  of  correctly 
identifying  persons  who  have  the  ability  to  perform  on  the  job  in  a  superior,  rather 
than  merely  in  a  satisfactory,  manner.   Usually,  these  criteria  are  different  from 
those  associated  with  satisfactory  performance 

d.  Methods  of  evaluating  candidates.   To  the  maximum  extent  possible,  the  same 
evaluation  methods  should  be  applied  to  all  persons  being  considered  for  appointment  to 
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higher  grade  positions.   When  that  is  not  possible,  for  example,  such  as  with  per- 
formances tests  where  candidates  may  not  be  in  the  same  geographical  area,  con 
sideration  should  be  given  to  either  not  applying  that  type  of  evaluation  method 
or  using  two  or  instruments  with  similar  relevance  and  validity  to  obtain  the  same 
type  of  information. 

e.   Grouping  candidates.   Evaluation  procedures  may  provide  for  the  ranking  of 
candidates  in  1-2-3  order  on  the  basis  of  scores  obtained  by  application  of  the 
evaluation  instruments  or  may  provide  for  assigning  candidates  to  quality  groups 
such  as  "qualified"  with  the  best  designated  for  referral  to  the  selecting  official. 
(See  section  B-ll). 

B-5.   STEPS  IN  DEVELOPMENT  OF  THE  EVALUATION  PROCESS 

The  steps  in  the  screening  and  evaluation  of  employees  should  be  designed  to  determine 
both  basic  eligibility  and  relative  qualifications  of  promotion  candidates.   In 
developing  the  process,  careful  consideration  should  be  given  to  each  of  the  follow- 
ing steps: 

a.  Determining  the  important  work  behaviors,  knowledges,  skills,  and  abilities,  and 
the  level  of  each,  that  are  needed  (1)  for  satisfactory  and  (2)  for  highly  success- 
ful job  performance. 

b.  Identifying  selective  factors:   (1)  are  they  appropriate;  (2)  are  there  any  other 
job  requirements  not  subject  to  evaluation  but  which  must  be  met.   (For  example  willing- 
ness to  travel  by  air.) 

c.  Determining  which  evaluation  instruments  should  be  used  by  finding  out: 

(1)  What  information  about  employees'  qualifications  is  already  available; 

(2)  What  additional  information  is  needed  to  determine  employees'  basic  qualifications; 

(3)  What  additional  information  is  needed  to  determine  the  best  qualified  employees; 
and 

(4)  Which  potential  instruments  (a)  are  relevant  to  the  job,  (b)  are  sound  and 
dependable  measures  of  the  qualifications  needed,  (c)  make  meaningful  distinctions  among 
the  employees,  and  (d)  are  practical  and  feasible.  (See  B-8,  Types  of  Evaluation  Methods) 

d.  Determining  specifically  for  each  instrument: 

(1)  At  what  stage  in  the  total  evaluation  process  it  will  be  applied; 

(2)  To  whom  it  will  be  applied: 

(3)  Who  will  apply  it;  and 

(4)  How  the  data  yielded  by  this  instrument  will  be  handled,  by  whom,  and  under  what 
instructions  for  interpretation. 

e.  Determining  how  the  data  or  results  of  each  instrument  are  to  be  combined  with  other 
information  in  ranking  employees,  and  how  they  are  to  be  used  in  the  final  selection 
decision. 

B-6.   DETERMINING  MINIMUM  REQUIREMENTS 

a.  General.   Minimum  job  requirements  are  in  the  qualification  standard  for  the  occupa- 
tion in  OPM  Handbook  X-118  and  X-118C .   The  standard  is  developed  on  the  basis  of  a 
comprehensive  occupational  analysis,  a  careful  study  to  identify  the  basic  and  (where 
required)  specialized  work  behaviors,  and  the  knowledges,  skills,  and  abilities  (job 
elements  or  factors)  needed  for  acceptable  performance  on  the  job.   Candidates  have 
basic  eligibility  for  the  position  when  they  satisfy  the  minimum  requirements  of  the 
standard  for  the  occupation  and  in  addition  any  specialized  requirements  (selective 
factors)  necessary  for  the  position. 

b.  Exceptions.   Employees  may  be  eligible  for  promotion  under  specific  provisions 
of  training  agreements  approved  by  the  OPM,  which  make  it  possible  to  substitute 
intensive,  accelerated  training  for  a  portion  of  the  normal  qualification  requirements 
(see  FPM  chapter  271,  subchapter  7). 
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B-7  NEED  FOR  ANALYSIS  OF  JOBS 

a.  Importance  of  job  analysis.   The  foundation  of  a  sound  evaluation  process  is  the 
careful  analysis  of  the  job.   This  critical  step  is  necessary  to  determine  minimum 
qualification  requirements,  to  identify  the  KSAds  to  be  used  to  identify  those 
candidates  who  can  be  expected  to  perform  in  a  superior  manner,  and  to  determine  the 
measurement  instruments  that  will  be  used  to  assess  qualifications  of  candidates, 
resulting  in  the  ranking  of  candidates  in  order  of  relative  ability.   The  design  of  the 
analysis  process  requires  personnel  measurement  skills.   Once  selected  and  designed, 
the  job  analysis  procedures  can  be  accomplished  by  subject  matter  experts  under  the 
guidance  of  carefully  trained  personnel  specialists.   A  complete  record  should  be 
retained  to  show  the  sources  of  job  data,  the  data  obtained,  and  a  statement  of  any 
conclusions  supported  by  the  data.   In  this  way,  a  logical  explanation  can  be  given 

to  support  and  justify  the  use  of  a  particular  evaluation  instrument  for  a  specific 
job  or  groups  of  jobs. 

b.  Scope  of  job  analysis.   (1)  Job  analysis  should  identify  not  only  the  work 
behaviors  and  knowledges,  skills,  abilities,  and  other  characteristics  needed 
for  successful  job  performance,  but  also  the  level  of  amount  of  KSAOs  needed.   On 
the  basis  of  this  information,  appropriate  measurement  methods  can  be  selected  or 
developed  in  terms  of  these  jobs  needs. 

(2)  One  way  to  utilize  the  information  about  job  requirements  gathered  from  a  job 
analysis  is  to  group  jobs  by  job  families,  i.e.,  jobs  having  similar  requirements. 
For  instance  a  variety  of  clerical  jobs  such  as  correspondence  clerk,  information  and 
editorial  clerk,  etc.,  might  be  considered  together  because  the  same  basic  requirements 
are  common  to  all  the  jobs.   Another  group  of  clerical  jobs  which  may  have  a  common 
core  of  qualifications  is  made  up  of  jobs  such  as:   retirement  clerk;  accounting  clerk; 
voucher  examining  clerk;  and  payroll  clerk.   These  jobs  may  have  in  common   accuracy 
and  speed  in  working  with  numbers,  attention  to  detail,  and  memory  for  a  variety 

of  items.   Thus,  the  employee  qualifications  needed  are  similar  and  one  evaluation 
program  might  suffice  for  promoting  employees  into  these  jobs  at  the  entrance  levels. 

(3)  Jobs  above  the  entry  level  will  usually  need  to  be  analyzed  separately  and  will 
probably  necessitate  separate  measurement  procedures,  since  the  work  behaviors  and 
KSAOs  which  differentiate  the  potentially  superior  employees  may  likely  be  different 
from  those  at  the  entry  level. 

(4)  The  job  analysis  should  also  include  information  about  the  total  environment 
of  the  job,  its  position  in  the  career  ladder,  and  its  training  features.   For  such  a 
job  analysis,  a  number  of  sources  of  information  can  be  utilized:   statements  of 
employees  and  their  supervisors;  official  position  descriptions;  review  of  work  materials 
and  work  products;  observation  of  work  processes;  job  performance  standards;  and  materials 
relating  to  other  personnel  and  operating  programs  and  policies  which  affect  this 
position. 

c.  Selective  factors.   If  an  element  is  found  to  be  essential  to  successful  perform- 
ance in  the  job  to  be  filled  and  is  not  covered  by  the  qualification  standard,  it 
may  be  added  as  a  selective  factor.   For  example,  a  position  in  a  particular  location 
may  require  knowledge  of  a  language  other  than  English.   Although  the  qualification 
standard  does  not  require  this  knowledge,  it  may  be  added  as  a  selective  factor. 

B-8.   TYPES  OF  EVALUATION  METHODS 

a.   A  variety  of  methods  are  available  for  evaluating  qualifications.   Among  these 
are  evaluation  of  training  and  experience,  written,  oral,  and  performance  tests,  inter- 
views, performance  appraisals,  and  assessment  centers.   Within  each  of  these  broad 
types  of  evaluation  methods,  there  are  various  specific  measuring  instruments  and 
procedures  that  may  be  applied.   These  include: 

(1)  Measures  of  demonstrations  of  specific  job  skills  such  as  typing; 

(2)  Measures  of  work  behaviors  or  KSAO's  demonstrated  in  structured  individual 
or  group  interviews  and  exercises  which  are  standardized  and  scored; 

(3)  Measures  of  abilities  related  to  learning  in  various  situations; 
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(4)  Measures  of  specific  classes  of  abilities;  verbal,  numerical,  mechanical, 
clerical,  and  others; 

(5)  Measures  of  dexterity  and  coordination; 

(6)  Measures  of  knowledge  and  proficiency; 

(7)  Occupational  and  other  interest  measures;  and 

(8)  All  other  formal,  scored,  quantified,  or  standardized  techniques  for  assessing  job 
qualifications. 

b.  Evaluation  of  training,  education  and  experience. 

(1)   Purpose.   An  evaluation  of  training  and  experience  is  used  to  predict  the  prob- 
able effectiveness  of  future  performance  of  the  basis  of  the  quality  of  relevant 
past  performance.   This  evaluation  should  cover  work  behaviors  or  knowledges,  skills, 
and  abilities  (the  outcomes  of  training,  education,  and  experience)  rather  than 
training,  education,  and  experience  for  their  own  sake.   Furthermore,  the  objective 
is  to  determine  how  well  the  candidate  is  prepared  for  higher  level  work.   Therefore, 
the  employee's  training,  education,  and  experience  should  be  evaluated  in  terms  of 
the  work  behaviors  or  knowledges,  skills,  and  abilities  needed  for  success  in  the  job 
to  be  filled.   (2)   Credit  for  training,  education,  and  experience.   Training,  education 
and  experience  are  evaluated  in  terms  of  the  degree  to  which  the  employee  possesses 
the  work  behaviors  or  the  knowledges,  skills  and  abilities  (job  elements  or  factors) 
identified  in  the  job  analysis.   For  each  factor,  various  degrees  of  ability  are 
described  in  terms  of  the  kinds  and  quality,  rather  than  length,  of  training,  education, 
and  experience.   These  descriptions  should  be  as  specific  as  possible;  care  should  be 
taken  not  to  rely  on  job  titles  since  different  employees  with  similar  job  titles  may 
be  performing  different  job  tasks  requiring  different  knowledges,  skills,  and  abilities. 
Normally,  it  is  sufficient  to  distinguish  four  to  six  levels  of  ability.   Greater 
refinement  is  not  usually  necessary  and  makes  the  descriptions  of  levels  as  well  as 
the  appraisal  of  employees  more  difficult  and  prone  to  error.   See  the  Job  Qualification. 
System  for  Trades  and  Labor  Occupations  (Handbook  X-118C)  for  examples  of  procedures 
for  crediting  training,  education,  and  experience  in  terms  of  job  elements.   Usually, 
supplemental  forms  are  needed  which  provide  candidates  the  opportunity  to  describe 
their  KSAOs  specifically  in  terms  of  their  relationship  to  job  requirements. 

(3)   Selection  of  evaluators.   Evaluation  of  training,  education,  and  experience 
requires  considerable  judgment;  the  employees  who  do  this  should  be  selected  carefully 
and  trained  well  in  the  use  of  the  rating  plan,  so  that  the  evaluations  by  different 
raters  will  be  comparable  and  dependable.   One  effective  approach  is  to  use  represen- 
tatives both  of  the  personnel  office  and  of  operating  offices  (subject-matter  experts) 
(SME's).   (See  section  B- 13  for  further  discussion  of  the  use  of  panels.)   Personnel 
specialists  can  contribute  a  knowledge  of  rating  techniques  as  well  as  of  career  staffing 
concepts  and  methods,  while  the  subject-matter  experts  can  assess  the  relative  value 
of  certain  types  of  experience  for  their  particular  jobs. 

c.  Employment  tests.   (1)  Purpose.   Professionally  prepared  written,  performance, 

or  oral  tests  can  provide  a  reliable  and  valid  means  for  measuring  knowledges,  skills, 
and  abilities  (KSAO's).   Tests  are  particularly  helpful: 

(a)  If  the  possession  of  specific  KSAO's  must  be  established;  for  example,  in  selecting 
people  for  jobs  which  are  new  because  of  technological  developments,  or  when  information 
about  the  KSAO's  of  employees  is  not  available; 

(b)  If  the  ability  to  learn  specified  new  job  behavior  or  concepts  is  an  important 
factor;  for  example,  in  evaluating  persons  for  trainee  positions  or  for  training  to 
qualify  for  higher  level  position; 

(c)  If  the  present  job  duties  of  the  employees  are  not  closely  related  to  those  of  the 
position  being  filled;  for  example ,  for  evaluating  potential  of  nonsupervisory  employees 
for  supervisory  positions; 

(d)  If  knowledge  of  important  subject  matter  is  to  be  evaluated; 

(e)  If  the  outcome  of  training  designed  to  prepare  employees  for  higher  level  positions 
is  to  be  evaluated;  or 
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(f)      If  relevant  knowledges,    skills,   and  abilities  among  the  employee   group  would  not 
otherwise  be  measured. 

(2)  Written  tests.      (a)      Written  tests   are  valuable   in  identifying  KSAO's   that  employ- 
ees may  have  had    little   opportunity   to   demonstrate   through   past   performance.      They  also 
are  appropriate  measures  of    information  or   knowledge   about  given   subjects. 

(b)  Written  tests  are   sometimes   in  the    form  of  questionnaires  designed  to   identify 
employee  occupational    interest  patterns.      These  questionnaires   are  usable   if  the   items 
are  relevant   to  the   specific  work  situations  or   tasks  and  thereby  enable  the  employee 

to   indicate  his  or   her   interest  in  the  particular    job.      Questionnaires  measuring   general 
traits,   such  as    social   interest,    usually  are  not  acceptable   for    selection  purposes. 

(c)  Written  questionnaires   intended  to  measure  personality  traits  are,   at  their 
present   stage   of  development,    generally  of  questionable  value  in   selection   of  personnel 
for  promotion.      FPM  chapter   337,    section   l-5i,   restricts  the  use   of  personality  tests   in 
personnel   actions   affecting  employees   in  the   competitive   service. 

(3)  Performance  tests.      Tests   in  which  the  employee  uses  physical   equipment  or   simul- 
ates or   replicates   a  work   process  or   product  are   commonly  labeled  performance  tests. 
Well-known   examples   are  the  typing  test  and  road  test  of  driving  skills.      Performance 
tests  are   particularly  useful   in  the   skilled  trades,   such  as  machinist  or   electrician. 
If   they  require  much  material   and  individual   examining,   administration  on  a   large   scale 
is   expensive.      On  the   other   hand,    properly  constructed  performance  tests  can   often 
provide  more   direct  measures  of   work  behaviors   and  KSAO's   than  written  tests.      Addi- 
tionally,  they  are  more   acceptable   to  employees. 

(4)  Oral   tests.      Oral   tests   are   standard  questions  which  are  used  to  measure   the 
employee's   ability  to  respond  to  problems  and  to   demonstrate   job  knowledge  by  oral 
responses  which   are  individually  scored.      Oral   tests  are  more  difficult  to   standardize 
and  more   costly  to   give  and  to    score  than  are   similar   written  tests. 

d.   Assessment  Centers. 

(1)  Purpose.      Frequently,   positions   are   filled  by  individuals  whose  present  or  previous 
job  requirements  are   somewhat  different   from  those   of  the  new  position.      The  assess- 
ment center   technique    is   a  valuable  tool  in  identifying   job-related  behaviors   and 

KSAO's  of  candidates  before  they  are  placed  on  the   job. 

(2)  Format.      The  assessment  center   consists  of   a  multiple  evaluation  procedure   in 
which  assessment  reports  are   developed  from  the   combined   judgments   of  a  team  of   trained 
assessors.      These  assessors   have   observed  the   candidates   in  various   standardized  and   job- 
related  simulated  situations.      An   advantage   of   assessment  programs   is   that  the   exercises 
can  be   tailored  to  the  organization  and   job  requirements.      Because   of   this   flexibility, 
there  are  a   number    of  variations   as   to  method. 

(3)  Content.      The  job  simulations  may  include   exercises   such  as  group  discussions, 
simulations   of   interviews   with  subordinates,   in-basket   simulations,  oral  presentation 
exercises,    written  communication  exercises,   and  factfinding  exercises.      The   exercises 
are   constructed  to  provide  the   candidate  an  opportunity  to   demonstrate   skills  that  have 
been   identified  as   important  for    success  on  the  job. 

(4)  Design  and  operation.      (a)      All  assessment  center    exercises   should  be   selected 
and  designed  to  measure   specific   job  requirements  as  determined  from   job  analysis.      As 
previously  referred  to,  the   simulations  must  be   job-related  and  standard  for    all   candi- 
dates who   are   competing  for   the   same   job(s).      In  addition,   those  work  behaviors  which 
are   to  be  observed  in  each  exercise   should  be   documented  and  specified  for   assessors. 

(b)      An   important  part  of   the  assessment  process   is   assessor   training.      The   assessors 
should  be   adequately  prepared   for   their    evaluative  task.     This  preparation   includes 
familiarization  with  the  assessment  exercises   and  the  relevant  skills,   training  on 
the  use   of  behavior  recording   forms  and  procedures,   and  instruction  as   to  the   final 
rating  process.      The  assessor   training  usually  requires  the   assessors   to  evaluate 
a  group  of   practice   candidates    which  provides  an   excellent  indication   of  rater  reli- 
ability.     Where   large  numbers  of   candidates  necessitates  multiple  teams  of   assessors, 
the  teams   should  be   equated  in  terms   of  rating  characteristics. 
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(c)   The  intended  use  of  assessment  reports  is  stipulated  in  advance,  and  the  candi- 
dates are  informed  of  the  assessment  center  procedures  and  objectives.   The  information 
obtained  during  the  assessment  process  may  be  appropriate  for  decisions  such  as  selection, 
placement,  or  career  development.   Candidate  feedback  is,  therefore,  an  additional  con- 
sideration.  This  feedback  should  focus  on  the  specific  skills  evaluated  during  the 
assessment  center.   Strengths  and  weaknesses  of  the  candidate  can  be  discussed,  and 
developmental  efforts  can  be  suggested. 

e.  Interviews. 

(1)  Purpose.   Some  important  elements  of  the  job  may  be  identified  from  the  job 
analysis  which  involve  personal  characteristics  of  the  employee.   In  some  cases,  observ- 
able information  pertinent  to  these  elements  cannot  be  obtained  adequately  by  means 
such  as  appraisals  by  supervisors.   In  these  instances,  an  interview  with  the  employee 
may  be  useful.   Interviews  should  of  course  be  job  related  and  be  used  consistently  with 
other  methods.   In  addition,  an  interview  may  be  useful  as  a  means  for  giving  information 
about  the  job  to  the  employee. 

(2)  Uses  in  ranking.   The  interview  may  be  used  in  the  ranking  process  in  two  ways: 

(a)  The  interviewers  evaluate  the  employee  on  specific  behaviors  and  responses  and 
this  information  is  later  combined  with  other  available  measures  to  arrive  at  a  final 
total  score.   In  this  way,  the  interview  information  is  simply  recorded  by  the  inter- 
viewers and  turned  over  to  those  making  the  final  evaluation.   The  interviewers  do  not 
combine  the  interview  results  with  results  obtained  on  other  measures. 

(b)  The  interviewers  evaluate  the  employee  on  specific  behaviors  and  responses 
considering  simultaneously  the  interview  data  as  well  as  other  available  information. 
in  this  second  way,  the  panel  of  interviews  must  combine  the  interview  results  with  the 
results  obtained  on  other  measures  to  arrive  at  the  final  evaluation. 

(3)  Planning  and  conducting  the  interview. 

(a)  It  is  critical  that  the  interview  be  well  planned  in  terms  of  the  behaviors  and 
responses  to  be  observed,  the  evaluation  standards  to  be  applied,  and  the  procedures  for 
conducting  the  interview  session.   It  is  particularly  important  to  pay  careful  attention 
to  objectivity.   The  interview  should  be  structured  so  that  the  information  to  be 
obtained  is  well  defined  and  recorded  in  a  standard  manner .   The  guide  for  determining 
employee's  score  on  the  factors  to  be  observed  should  specifically  state  how  the  informa- 
tion from  the  interview  is  combined  with  information  from  other  sources. 

(b)  In  addition  to  the  careful  design  of  the  interview,  there  should  be  at  least  two 
interviewers  who  are  well  trained  in  the  procedures  and  the  purpose  of  the  interview. 

(c)  Since  the  interview  at  best  is  only  a  small  sample  of  the  employee's  behavior 
and  is  conducted  in  a  setting  which  may  not  be  representative  of  the  work  situation, 
care  should  be  taken  not  to  attribute  more  precision  to  the  interview  than  it  warrants. 
Small  differences  among  employees  are  unlikely  to  be  meaningful. 

(d)  Interview  records  should  be  retained  with  other  merit  promotion  plan  records. 

(e)  If  no  work  behaviors  or  KSAO's  are  identified  that  must  be  measured  by  an  inter- 
view, it  is  not  necessary  to  interview  the  candidates.   However,  if  personal  contact 

is  desired,  a  selection  interview  may  be  held  in  accordance  with  FPM  chapter  332,  Appendix 
I. 

(f)  Interviewers  should  take  special  care  not  to  ask  questions  which  violate  equal 
opportunity  law. 

f.  Appraisals  of  work  behaviors,  knowledges,  skills,  abilities  and  other  characteristics. 
(1)   Purpose.   (a)   Appraisals  of  employee  performance  are  used  for  various  purposes, 

e.g.,  promotion,  employee  development,  and  performance  ratings  required  by  statute.   A 
principal  difference  between  appraisals  for  promotion  purposes  and  appraisals  for  other 
purposes  is  that  appraisals  for  promotion  focus  on  the  job  to  be  filled,  and  appraisals 
for  other  purposes  focus  on  the  employee's  present  job.   Both  the  orientation  of  the 
rater  and  the  content  of  the  appraisal  instrument  frequently  differ  according  to  the 
purpose  of  the  appraisal.   It  is  important  that  appraisals  used  to  evaluate  employees 
for  promotion  be  relevant  to  the  job  for  which  they  are  being  considered. 
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(b)      Frequently,  employees   have   not  had  an  opportunity  to   demonstrate   some   of   the 
important  work  behaviors  or   KSACfe  needed   for   the   job  to  be   filled.      For   work  behaviors 
and  KSAds   which   differ   markedly   from  those  which   are   needed   to  perform  the    candidates' 
present   jobs,    performance   appraisals  may  not   offer   useful   information    for    evaluation 
of    those  work  behaviors   and  KSACfe.      However,    care   should  be   taken   in   the   design   of 
appraisal   procedures    to  ensure   that   candidates   get  due    credit    for   related  work  beha- 
viors   and  KSAtfs;    for   work  behaviors    and  KSACfe   which  have   been   demonstrated   at    slightly 
different   levels   than   those   needed  on   the   new  job;    and   for   work  behaviors   and  KSAds 
which   have    been   demonstrated   in  other   ways,    such   as   training  programs   or    temporary 
assignments.      It   is  usually  advisable   to   supplement   appraisals   by   supervisors   with 
information   about  work  behaviors   and  KSAC/s   from  other    sources,    including  the    candi- 
dates   themselves. 

(2)  Periodic   appraisals   of   performance.      Some   agencies   confronted   with   the   need 
to    evaluate    large    numbers   of   employees   have    developed  systems    in   which   appraisals 
are  obtained  at  regular   intervals.      Some   of   these   systems   are   computer   based  and 
cover    a  wide  range    of   work  behaviors  or    factors   designed  to  record  information    for 
use   in   appraising   the   pr omotability  of   employees   for    a   large   number    of   different 
positions.      To   develop  an   appraisal    score,  a   subset  of   factors  that  are  relevant 
to   the  job  to  be   filled  is   selected  from  the   large  number   of   factors  rated.      If 
relevant   factors    for   a  particular    job  are  not  included  in  the   periodic  appraisal, 
the  periodic  appraisal   should  be   supplemented  with  special   appraisals.      Similarly, 
if  the   periodic   appraisal   includes   factors   which  are  not  relevant   to  the   job  to  be 
filled,   the  ratings   on   these   nonrelevant   factors    should  not   be   used    in   determining 
the    appraisal    score. 

(3)  Format.      Often  one  of   two   forms- -narrative  or   check-off — is  used.      Each   form 
has    its    advantages   and    disadvantages.      The    check-off  may  provide   the   supervisor    less 
freedom   in  reporting  his   or   her   observations   of   the   employee's   behavior.      However, 
since  the   check-off   is  more   structured,   it   is  more    likely  to  have   satisfactory 
objectivity   and  reliability.      Frequently,   a    combination   of    the   two   is   used   in  which 
the   supervisor    first  completes  the   check-off  portion  and  then  supports  his  or   her 
judgments   with   narrative    statements.      The   forms    should  encourage   a   spread   of    scores. 
This   can   be    facilitated  by   identifying  the   various   points  on   the    scale    for    each   factor 
in   terms  of    specific   behaviors,    and  requiring  the   supervisor    to  report   his   or   her 
observations  rather    than  make    a  judgment   of    whether   the   employee   is   high   or    low. 

(4)  Improving   consistency  of    appraisal    and   assessment.      When   the   employees    are 
scattered  throughout   a    large   organization,   they  will   be   evaluated  by  different   people. 
The  various   people  may  differ    in   their   understanding  of   the   job  requirements   and   the 
factors    to   be   covered.      Use    of   the    following  procedures  may  help    to  provide   consistency 
of    judgments   when   different  people   evaluate   the   employee. 

(a)  The   factors  used  are   based  on   careful  analysis  of   job  requirements   in  order   to 
reach    a   common   understanding. 

(b)  Evaluators,    when  possible,    discuss   the    factors   and   job  requirements    (but   not 
individual    employees)    in   order    to  reach  a    common   understanding. 

(c)  Supervisory   judgments   are   obtained   in  a   systematic  way   so   that  work   in   different 
situations    under    different   people   can   be    evaluated  in   a   standard  manner. 

(d)  Training   is  given   to   ensure   that,    before   they   start   to  make   the  evaluations,    the 
evaluators   are    given   careful   instructions.      Also,   the   effects   of   their    evaluations 

on  quality    staffing   as   well   as  on   the   careers   and  the   attitudes   of   employees   are  made 
clear . 

(e)  Results  are  obtained  from  several   independent  evaluations,    if  appropriate  to 
furnish    a  more   balanced  picture   than   a   single   appraisal.      In   addition,    knowledge   of 
the   fact   that  future   opportunities  will  not  be   determined  by  just  one   supervisor's 
judgment    is   likely   to   increase   the   employee's    acceptance   of   a   promotion   plan. 

B-9.       DETERMINING   EFFECTIVENESS   OF    INSTRUMENTS 

a.      General.      The  major    consideration   in    selecting  and  using  an   evaluation   instrument 
is    its   effectiveness    in  measuring  the   degree   to  which   the   employees   possess   the  work 
behaviors   and  knowledges,    skills,   and   abilities   needed   for    successful    job  performance. 
The  effectiveness  of   any  instrument  depends  on  its/  job  relatedness,   objectivity, 
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reliability,  and  validity.   These  qualities,  individually  and  in  interaction,  are  critical 
to  a  sound  program  of  identifying  persons  best  able  to  perform  or  learn  a  job.   Each 
of  these  qualities  is  defined  in  general  terms  below  and  should  be  considered  by 
persons  responsible  for  selecting  and  applying  evaluation  instruments. 

b.  Job  relatedness.   An  evaluation  instrument  is  job  related  if  the  work  behaviors  or 
knowledges,  skills,  abilities,  and  other  characteristics  (KSAO's)  measured  by  it  have 
been  determined  to  be  necessary  for  successful  job  performance  through  a  job  analysis. 
For  example,  a  supervisory  appraisal  of  performance  is  relevant  only  to  the  extent 
that  the  work  behaviors  or  KSAO's  being  rated  are  important  for  success  on  the  job 

to  be  filled.   If  accuracy  in  working  a  calculator  is  not  important  in  a  supervisory 
statistician's  job,  appraisals  of  that  skill  are  not  relevant  in  evaluating  employees 
for  the  supervisory  job.   The  need  for  job  relatedness  again  points  up  the  need  for 
careful  job  analysis.   Instruments  are  selected,  modified,  or  developed  on  the  basis 
of  the  findings  of  the  job  analysis  to  assure  they  measure  important  job  characteristics. 
This  helps  assure  that  all  qualified  employees  receive  proper  consideration. 

c.  Objectivity.   An  evaluation  instrument  should  be  as  objective  as  possible; 

that  is,  it  should  be  structured  to  elicit  responses  which  are  factual  and  observable, 
and  which  are  reported  in  a  precise,  specified  way.   In  addition,  specific  rules  should 
be  available  for  determining  the  value  of  each  response.   For  example,  in  evaluating 
employees  for  computer  operator  positions,  it  would  be  more  objective  to  ask  them  to 
list  the  computers  they  had  operated,  and  the  kinds  of  operations  they  performed  on 
each  rather  than  to  try  to  make  judgments  based  on  general  descriptions  of  duties. 
Evaluation  of  this  experience  can  then  be  systematized.   The  machines  and  tasks 
indicative  of  the  highest  level  of  ability  can  be  specified  and  each  lower  level  of 
ability  can  be  defined  in  similar  terms.   The  less  objective  a  measurement  procedure 
is,  the  greater  the  possibility  of  unreliability. 

d.  Reliability.   Reliability  refers  to  the  consistency  and  stability  of  the  scores 
obtained  with  the  instrument.   Any  measurement  is  subject  to  error  and  the  extent  to 
which  repeated  measurements  differ  limits  the  reliability  of  the  measuring  instrument. 
Reliability  can  be  estimated  in  a  number  of  ways:   On  tests,  for  example,  by  comparison 
of  results  of  repeated  administrations  of  the  same  test,  on  parallel  forms,  or  on 
representative  subsets  of  items;  on  appraisals,  by  comparison  of  the  ratings  assigned 
by  different  appraisers.   The  more  reliable  an  instrument,  the  less  likely  it  is 

that  a  person's  score  is  affected  by  chance  errors.   It  should  be  noted  that  an 
instrument  can  be  reliable,  but  invalid. 

e.  Validity.   An  instrument  is  valid  if  it  measures  the  work  behaviors  or  knowledges, 
skills,  abilities  and  other  characteristics  (KSAO's)  it  is  intended  to  measure.   Vali- 
dity is  a  critical  requirement  for  the  selection  and  use  of  any  measuring  instrument. 
However ,  an  instrument  may  provide  a  highly  valid  measure  of  a  KSAO  and  still  not  be 
appropriate  for  a  particular  job  because  the  KSAO  itself  is  not  related  to  successful 
job  performance.   Thus,  both  validity  and  job  -  relatedness  should  be  considered.   There 
are  three  generally  accepted  ways  of  demonstrating  validity:   content,  construct,  and 
criterion  related.   At  least  one  form  of  validity  evidence  is  normally  required  to  sup- 
port the  use  of  an  instrument.   Principal  concepts  of  these  approaches  are  provided 
below  in  a  highly  simplified  manner.   Further  guidance  on  validation  and  documentation 
is  covered  in  Appendix  A.   (1)   Content  validity  depends  on  well-informed  judgments  that 
the  content  of  the  measuring  instrument  is  representative  of  important  aspects  of  perfor- 
mance on  the  job  (see  paragraph  14C  of  Appendix  A).   Job  information  tests  and  other 
achievement  tests  are  typically  evaluated  on  the  basis  of  content  validity.   Supervisory 
appraisals,  ratings  of  training  and  experience,  and  assessment  centers  are  also  frequently 
evaluated  on  the  basis  of  content  validity.   In  the  case  of  evaluation  of  training  and 
experience  where  information  provided  by  candidates  is  evaluated  in  terms  of  behavioral 
examples  of  demonstrated  KSAOs ,  it  is  important  to  content  validity  that  the  candidates 

be  provided  a  description  of  the  important  work  behaviors  or  KSAO's  reeded  on  the  job  so 
that  they  can  select  appropriate  examples  from  their  experience.   In  such  cases,  the 
candidates  sample  from  their  experience  those  accomplishments  and  behavioral  examples 
which  most  closely  match  the  KSACfe  required  on  the  job. 
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(2)  Construct  validity  is  based  on  the  inference  that  a  certain  measurable  ability 
or  personal  characteristic  (construct)  is  required  in  the  job  and  that  the  selection 
procedure  in  fact  measures  this  construct.   For  example,  a  job  analysis  may  indicate 
that  quantitative  reasoning  ability  is  important  for  successful  job  performance.   A 
test  is  then  devised  or  selected  to  measure  the  quantitative  reasoning  ability.   If 
the  test  can  be  shown  to  be  related  to  important  job  duties  it  may  be  used  opera- 
tionally.  Identification  and  measurement  of  constructs  is  an  involved  process. 
Again,  job  analysis  is  an  essential  part  of  the  process,  and  usually,  evidence  of 
the  relationship  of  the  measure  of  the  construct  with  other  measures  of  the  same 
construct  is  necessary  for  demonstration  of  construct  validity. 

(3)  Criterion-related  validity  is  determined  by  statistically  comparing  performance 
on  the  measuring  instrument  with  some  other  indicator  (the  criterion)  which  is  accepted 
as  a  measure  of  job  performance.   Criterion-related  validity  can  be  approached  in  two 
ways:   predictive  validity  and  concurrent  validity.   Predictive  validity  involves 
relating  scores  on  a  measuring  instrument  to  later  measures  of  job  performance.   For 
example,  scores  on  an  ability  test  might  be  used  to  predict  later  success  in  job  per- 
formance.  In  concurrent  validity,  the  criterion  score  is  obtained  at  approximately 
the  same  time  as  the  score  on  the  evaluation  instrument.   For  example,  a  test  of 
verbal  ability  might  be  administered  to  a  group  of  employees  and  compared  with 

their  present  job  performance. 

f.   General  standards  for  validation.   An  evaluation  instrument  is  acceptable  if  either 

of  two  sets  of  conditions  is  met: 

(1)  There  is  evidence  from  a  careful  job  analysis  that  the  work  behaviors  or  KSAO's 
being  measured  are  in  fact  an  important  part  of  the  job  (class  of  jobs,  or  occupation) 
and  logically  related  to  successful  performance  on  that  job  (class  of  jobs  or  occupa- 
tion); and  there  is  competent  evidence  of  content  or  construct  validity  in  the  evalua- 
tion instrument  or   (2)  There  is  evidence  of  a  useful  degree  of  criterion-related 
validity  arrived  at  by  comparing  scores  on  the  instrument  with  a  criterion  which  is 
legitimately  based  on  the  needs  of  the  Federal  Government. 

B-10.   OTHER  CONSIDERATIONS  IN  SELECTING  INSTRUMENTS 

a.  General.   Many  factors  enter  into  the  decision  on  what  instruments  should  be  used 
in  the  evaluation  process.   There  are  many  administrative  considerations.   The  primary 
consideration,  however,  is  obviously  the  effectiveness  with  which  the  instrument  can 
provide  adequate  measure  of  the  work  behaviors,  knowledges,  skills,  abilities  or  other 
characteristics  (KSAO's)  required. 

b.  Differences  between  selection  for  entry  and  for  inservice  placement.   (1)   In 
selection  from  among  applicants  outside  the  agency,  the  selecting  official  chooses  from 
among  many  persons  not  directly  known  to  him  or  her.   In  inservice  actions,  however, 
the  agency  also  must  recognize  the  existing  employment  relationship  and  the  nature  of 
the  commitments  between  the  employees  and  the  agency.   The  methods  used  help  determine 
whether  the  employee  will  consider  the  organization  fair  in  its  treatment  of  employees. 

(2)  In  making  promotion  decisions,  the  agency  is  likely  to  have  available  more  com- 
plete and  dependable  information  about  its  employees  than  it  is  likely  to  have  about 
outside  applicants.   Therefore,  it  can  and  should  use  evaluation  instruments  and  proce- 
dures which  make  full  use  of  available  information  relevant  to  evaluation  of  specific 
work  behaviors  or  KSAO's  required  for  particular  positions. 

c.  Effect  of  new  ,r   modified  instruments.   (1)   The  introduction  of  new  or  modified 
measuring  instruments  calls  fot  careful  study  of  the  agency's  staffing  function  and  of 
selection  factors  already  beiny  used.   It  is  important  to  find  out  what  is  already  known 
about  the  work  behaviors  and  knowledges,  skills,  and  abilities  of  employees  in  various 
occupations  to  see  whether  new  or  modified  instruments  are  needed.   What  selection 
devices  h^ve  already  been  used,  and  at  what  level  of  difficulty?   How  narrow  or  how 
broad  is  the  r^nge  of  individual  differences  among  present  employees  as  a  res     f 
high  selection  standards  for  intial  app   ntment,  of  turnover,  and  of  training?   Employ- 
ment standards  and  procedures  in  the  Fedor.1  Government  have  a  very  marked  selective 
effect.   The  parsons  from  the  total  employment  population  who  are  hired  represent  a 
selected  yroup  both  in  terms  of  general  and  special  KSAO's. 
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This   selectivity  means   that   a  more  restricted   range   of    individual    differences    exists   among 
employees   than    among  applicants.      Thus,    it   becomes   more    difficult    to    identify   large    differ- 
ences   among  qualified   employees   than   between  qualified   and   unqualified   applicants    in 
open,    competitive    examinations. 

(2)      The    instruments   and  procedures   under    consideration,    if   otherwise   technically   sound, 
should   meet   the   test    of   acceptability  and  reasonableness    to   the   employees    affected  and 
the   supervisors    and   other    officials   who  will   be  making  use   of    the   evaluations    for   reach- 
ing personnel    decisions. 

d.      Administrative    considerations.      Many   problems   of   an  administrative   nature   will   affect 
the    choice   and  use    of    specific   instruments.      Although   these   administrative    problems  must 
be    considered,   the  main   goal--to  provide   a   sound  promotion   program — should  not   be   over- 
looked.     In    some   cases,    the    decision    to   select  one   instrument   over    another   may  result   in 
the   need   to  put  more   time    and   effort   into   the   evaluation   of   employees,    but   this   additional 
workload  may  be  worthwhile   in   terms   of   increased   effectiveness   of   the  method,    employee 
confidence   in   the   program,    etc.      The   administrative   considerations    include    such  matters 
as   the   number    of   employees    to   be    considered   in  relation    to   cost   of   program,    number    of 
vacancies,    etc;    whether    the  program   is  designed   for    large   groups    in  populous   occupational 
areas   and    grade    levels,   or    for    small    groups   in    specialized  positions,   or    both. 

B-ll   GENERAL    PRINCIPLES   ON   ARRANGING   EMPLOYEES    IN   ORDER   OF    MERIT 

a.  Introduction.      To    identify   the   best   qualified   employees,   the   results   of   evaluation 
instruments   should  be   used   to   rank   or    group  eligible   candidates    in   order    of  relative 
merit.      Such   an   arrangement   can  range    from  broad   categories,    each   including  a   sizable 
group  of    employees,    to   specific  rankings.      Two    interrelated  problems   are   involved:      First 
how  to   combine   the   results    of   several    instruments;    second,    whether    the    score   produced 

by  the   combination    is   suitable    for    fine  ranking  or    for    broad   categorization. 

b.  Principles.       (1)      The    degree    of  refinement   expected  should  not   exceed  the    sensiti- 
vity of    the  measurement    instrument   itself.      For    example,    interviewers   or    supervisors  may 
be    able   reasonably   to   appraise   various   dimensions   of   employee   performance   on   a   broad 
5-point    scale.      If,    however,    the   procedure   requires   them  to  use   a   highly  refined   scale 
with   a    large    number    of   different   divisions,    small    differences   in   the   appraisals   of 
employees   will    likely  reflect   only  apparent  rather    than  real    differences    in   ability. 

(2)  It   is    important   that   instruments   and  procedures   intended  to  measure   work 
behaviors   or   KSAO"s   which   differentiate    superior    employees    from   less    successful 
employees    actually   differentiate    among   employees    when   applied.      If   almost    all    candi- 
dates get   the  maximum  available   points   of   evaluation   of   performance,    for    example, 
other    possibly  less    important    factors  may  then   account    for   the  rankings   of   candidates. 
When  work  behaviors   or   KSAO^s   are   identified   in   job  analysis   which   distinguish   superior 
performers,   and   attempts   at  measurement    of   these   particular   work  behavior    or    KSAds 
fail    to   demonstrate   differences,    the    job  analysis   should  be  reviewed  with  respect 

to   the    appropriateness    of   the   work  behaviors   or    KSAOfe    identified.      If   their    appro- 
priateness   is   confirmed,    it   then   becomes   appropriate   to   determine  ways   to    improve 
the  measurement  methods   and  procedures    to   obtain    a  more   accurate    spread   of  measures 
on   employees   and   valid   ranking. 

(3)  The  person  who  makes  the  final  selection  normally  should  not  be  confronted  with 
so  many  employees  that  he  or  she  cannot  reasonably  review  every  case  and  make  compari- 
sons   among   all   the   employees    in   the   best   qualified   category.      At   the    same   time,    it 

is    improper    to  reduce   the   number    in   the   best   qualified   group  on   the   basis   of  minute 
score   differences,    or    arbitrary  or    artificial    factors,    which   are   unlikely  to   have 
real   meaning  in   terms   of   likelihood   of    success  on   the    job.      Where    too    large   a   number 
of    employees    in   the   best  qualified   group  continues   to   be   a  problem,    it   is  probably 
indicative    of  a   need   to  review  the    job   information   and   selection   procedure   to   focus 
more    clearly  on  work  behaviors   or    KSACJs   which   distinguish  highly   successful   performance. 
Where    distinctions    simply  cannot   be  made   tie-breaking  factors   such   as   length  of 
experience   or    seniority  may  be   used. 
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(4)  In  all  cases,  even  when  only  a  few  employees  are  under  considerations,  the  results 
obtained  by  the  employees  on  the  evaluation  instruments  are  of  direct  value  in  describing 
the  relative  abilities  of  the  employees  for  the  selecting  official. 

B-12   COMBINING  RESULTS  OF  VARIOUS  INSTRUMENTS 

a.  Successive  hurdles.   Combining  measurement  results  is  one  of  the  most  important 
functions  of  the  professional  personnel  measurement  specialist.   Among  the  considera- 
tion faced  by  a  measurement  specialist  is  a  procedure  by  which  the  various  evaluation 
instruments  are  used  as  a  sequence  of  hurdles.   Under  such  a  system,  the  employee 
must  attain  a  certain  score  on  each  instrument  as  that  instrument  is  applied.   Only 
those  who  attain  the  specified  score  on  each  instrument  are  given  final  consideration. 
This  procedure  may  be  appropriate  in  situations  in  which  a  particular  work  behavior 

or  KSAO,  or  a  specific  level  of  work  behavior  or  KSAO,  is  necessary,  in  and  of  itself, 
for  successful  performance  on  the  job;  and  absence  of  it  cannot  reasonably  be  compen- 
sated for  by  other  KSAOs.   Hurdles  or  screening  factors,  are  usually  appropriate 
only  in  determining  basic  eligibility.   In  distinguishing  among  the  qualified  and 
the  best  qualified  candidates,  the  interests  of  the  employee  and  the  agency  are  best 
served  by  basing  the  employee's  evaluation  on  the  total  picture  drawn  from  a  variety 
of  evaluation  instruments.   In  this  way,  employees  who  are  unable  to  show  their  ability 
on  one  instrument  might  be  able  to  show  it  on  another. 

b.  Methods    of  combining   scores. 

(1)  The  combining  process  is  difficult  when  the  results  on  each  of  two  or  more 
evaluation  instruments  are  to  affect  the  rank  ordering  of  the  employees.   The  various 
scores  must  be  combined  to  yield  a  single  final  score.   There  are  various  ways  in  which 
evaluation  data  can  be  combined.  The  particular  way  selected  determines  which 
evaluation  instrument  has  the  greatest  effect  on  the  final  ranking. 

(2)  The  weighting  of  scores  and  ratings  is  not  a  mechanical  process.   Sometimes  when 
weights  have  been  set  and  scores  have  been  multiplied  by  the  weights,  the  real  influence 
of  the  weighted  scores  may  be  vastly  different  from  the  apparent  or  intended  one  because 
of  the  statistical  properties  of  the  measuring  instruments.   When  information  from  several 
different  measuring  instruments  is  to  be  considered,  the  information  is  often  expressed 
in  different  terms  for  the  different  measures.   For  example,  supervisory  appraisals  may 

be  in  terms  of  descriptions  of  levels  of  performances,  whereas  test  results  may  be  in 
terms  of  numerical  scores.   Morever,  the  terms  of  evaluation  may  be  of  dissimilar  magni- 
tude, with  supervisory  appraisals  ranging  from  one  to  five  levels  and  test  scores  ranging 
from  50  to  100.   These  differences  affect  the  final  ranking.   Raw  scores  on  a  written 
tests  and  on  an  evaluation  of  training  and  experience,  for  example,  cannot  simply  be 
added  together  with  the  thought  that  equal  weight  is  being  given  to  the  two  parts.   Since 
the  written  test  scores  are  likely  to  be  spread  out  over  a  much  wider  range  than  were 
the  ratings  of  experience,  the  written  test  will  therefore,  have  greater  weight  in  the 
composite.   Before  deciding  upon  how  to  combine  scores  from  measuring  instruments,  it  is, 
therefore,  essential  to  know  their  real  effects. 

(3)  One  of  the  best  ways  of  controlling  the  effects  of  differences  between  the  terms 
used  for  different  evaluation  instruments  and  thereby  making  interpretation,  comparison, 
and  combination  of  data  from  various  sources  easier  is  to  convert  the  raw  scores  on  each 
instrument  to  standard  scores.  The  procedure  for  computing  standard  scores  is  explained 
in  tests  on  psychological  statistics.  This  results  in  scores  on  one  instrument  that  can 
be  directly  compared  or  combined  with  scores  on  another.  For  the  OPM  tests  covered  in 
this  appendix,  standard  scores  have  been  computed  and  are  included  in  the  discussion 

of  the  application  of  scores.   Agencies  may  find  this  information  useful  as  an  aid  in 
interpreting  scores  on  these  tests. 

c.  Relating  information  to  elements  rather  than  total  job.   (1)   In  some  cases,  the  score 
obtained  on  each  evaluation  instrument  is  considered  as  a  separate,  overall  measure  of 
the  employee's  potential  for  success  on  the  job  and  the  results  of  the  various  measures 
are  then  combined  into  a  single  total  score.   It  may  be  more  appropriate,  however,  to 
view  the  results  of  the  various  instruments  as  alternate  indications  of  ability  on  each  of 
the  job  elements  (KSAO's)  to  which  the  information  is  related.   A  scoring  plan  can  be  set 
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up   for    each    element   to   consider    all   the   pieces   of    information   that   are   indications   of 
ability   in   that    element  regardless    of   the    evaluation   instrument   that   produced  the   infor- 
mation.     In   this  way,   all   the  available   information   is  used   to  arrive   at  the   separate 
element    score.      (The   key  is  that  the   plan  must   specifically  state   how  the  various  pieces 
of    information  are  used   to   determine  the   score  on  the   element.)      Then,   the   scores  on 
the    elements   are   combined   to    form  an   overall    indication    of   how  good  the   employee    should 
be   on   the   job. 

(2)      This   technique    is   used    in   the   job   element  method  described   in   the   Job  Qualification 
System   for   Trades   and  Labor    Occupations    (Handbook   X-118C) .      In   this  method  a    logical 
general    scoring   system   is  provided  to   assure   that  the    information  relating  to   each    element 
is   looked   at   in   a   consistent   way.      Since    the  method  uses   only  those    few  elements    which   are 
most   effective    in    identifying  the   employees   who   are  most   likely  to   be    successful    on   the 
job,   the    scores  on  the   elements   are   given   equal  weight   unless   evidence   is   available   to 
support   assigning   a   higher   weight   to    specific   elements. 

B-13.       USING    PROMOTION    PANELS 

A  promotion  panel    is   a  permanent   or    ad   hoc   committee   established  to   evaluate,    compare, 
and  rank  employees    for    promotion.      The  membership  of  the   panel  normally  consists   of   three 
to    seven   agency  program   or    operating   officials    and  personnel    office   representatives.      At 
least  one   personnel   office  representative   should  be  in  attendance  at  the  panel  meetings, 
so   that  control   and  documentation  of   panel   actions  meet  validity  requirements.      Usually, 
some    of   the    line    officials   on   the   panel    are  representatives   of   the   organizations   in 
which   the  vacancy  exists.      To  assure  objectivity  the   selecting  official   should  not  be 
a   panel  member.      In   exceptional  situations  his  or   her    expertise  and  knowledge   of 
the  position  being  filled  may  be  essential    for   the   effective   evaluation  of   the   candidates. 
In   such  cases   it  would  be   advisable    for   the  personnel   office  representative   to   document 
the  reasons   for   his  or   her   inclusion  on   the  panel.      This   situation   calls   for    a  very 
clear    understanding   of   the    functions   of   the   panel   members   in   the    evaluation   process 
as  distinct   from  the   selection   decision  to  be  made  by    the   selecting  official.      If 
other    officials   are    available   who   have   a   sufficient   knowledge    of   the    job   to   be    filled, 
such    individuals    should  be   included   on   the   panel   rather   than   the    selecting   official. 
The   procedures    to  be    followed  by  the   panel,   the  way  results  of   evaluation   instruments 
are    interpreted   and  used,    and  the  method   for    combining  results   to   arrive   at   the    final 
ranking  should  be    precisely  stated  and  clearly  understood  by  the   panel   members. 

B-14.       DETEFMINING   EFFECTIVENESS   OF    THE   EVALUATION   PROCESS 

a.  Need   for   review.      After   the   evaluation   process  has  been  initially  developed  or    after 
it  has  been  used  to  place  employees  on  the   job,   the  question   arises,    "Are   there   addi- 
tional  ways   in   which  the   effectiveness   of  the   process  may  be   approved?"      Any  information 
that  indicates  deficiencies  or    suggests   improvements   is  of  value   in  refining  the  process 
so  that   it  results   in   identification   of   the   best  qualified  employees. 

b.  Tryout   of   evaluation  process.      One  procedure   is  to  try  out   the  process  with  employees 
of   known   competence,    so  that   instruments   and  procedures  can   be   improved  before   they  are 
actually  used   in  the   evaluation  process.      In  many  cases,   however,   the   conditions   neces- 
sary to  make   a    sound  pretest   of  this   kind  do  not   exist.      Frequently  there  are   too   few 
employees   who   are  working    in   similar   positions    and   for   whom  meaningful    evidence   of    job 
performance    can   be   obtained.      Another    limitation   is   that,    since   employees    already  on   the 
job  represent   a  highly   selected    group,    the   narrowed  range   of    ability   within   the    employed 
group  makes   it  difficult   to  relate   differences   in   job  performance   to  measurable   diffe- 
rences  in  knowledge,    skill,   and   ability. 

c.  Placement   followup.      A  practicable  method  of   determining  how  to   improve   the  evalua- 
tion  process   is    to   see   how  well  people  who  were   promoted  are   doing  on  the    job  after   6 
months  or   a  year.      Feedback  of   this  type   can  be  useful   for    identifying  possible  weaknesses 
in   evaluation   procedures. 
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(1)  If  the  employees  promoted  are  successful  on  the  job,  it  can  generally  be 
concluded  that  the  methods  of  selection  has  been  sound  and  that  they  have  contributed 
to  the  selection  process. 

(2)  One  rough  guide  is  the  number  of  promotees  who  had  to  be  reassigned  or  given 
special  training  (other  than  training  normally  planned  or  expected)  because  of  inability 
to  perform  the  duties  of  the  job.   Analysis  of  the  particular  deficiencies  provides  a 
basis  for  improvement  of  the  evaluation  process.   Did  the  evaluation  process  include 
some  instrument  which  was  supposed  to  guard  against  this  particular  deficiency  and 
apparently  did  not  function  as  intended?   Should  techniques  be  added  to  cover  this 
qualification  requirement?   Frequently,  careful  analysis  of  the  reasons  for  success  or 
failure  of  particular  persons  may  provide  insights  into  the  nature  of  the  entire  promo- 
tion plan  and  directions  of  improvements. 

d.  Other  benefits.   Observation  and  employee  feedback  provide  a  means  to  check  up  on 
other  aspects  of  the  promotion  program,  e.g.,  the  usefulness  and  effect  of  orientation 
or  similar  training  efforts;  the  adjustment  (as  distinguished  from  performance)  of 
employees  in  their  work  and  its  environment;  and  the  reactions  of  employees  and  super- 
visors to  the  promotion  program.   From  this  type  of  information  it  should  be  possible, 
over  a  period  of  time,  to  develop  an  appreciation  of  the  real  effect  and  value  of  the 
promotion  plans  in  the  agency's  overall  personnel  management  system,  and  the  problems 
which  need  attention. 

B-15.   INFORMATION  ON  WIDELY  USED  OFFICE  OF  PERSONNEL  MANAGEMENT  EVALUATION  PROCEDURES 

a.   General.   The  procedures  listed  in  this  section  may  be  used  by  any  agency.   They 
must  be  used  in  accordance  with  the  occupational  and  grade  level  coverage  and  any  other 
restrictions  prescribed  in  the  considerations  for  use  of  each  test  or  other  procedure. 
(When  a  test  is  authorized  for  use  at  more  than  one  grade,  however,  it  is  not  the  intent 
of  the  OPM  that  it  be  applied  more  than  once  to  the  same  employees.   If  the  test  is  used 
for  selecting  employees  at  one  level  within  an  occupation,  it  may  not  then  be  used  for 
evaluating  these  employees  for  promotion  to  a  higher  grade  in  the  same  or  related  occupa- 
tion.)  The  test  must  be  properly  administered  and  scored.   Information  to  assist 
agencies  in  interpreting  and  applying  scores  is  provided  for  use  in  combining  the 
test  results  with  results  of  other  procedures  for  which  standard  scores  are  available. 
(See  Appendix  B-12). 
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b.   Clerical  abilities  test.   (1)   Introduction.   The  clerical  abilities  test  includes 
measures  of  (a)  clerical  abilities  (Test  800)  and  (b)  verbal  abilities  (Test  801A) . 

(2)  Considerations  for  use. 

(a)  0PM  Tests  800  and  801A  may  be  used  by  agencies  for  those  GS-2  and  GS-3  clerical 
positions  for  which  the  tests  may  be  used  competitively.  A  determination  must  be  made 
that  the  abilities  measured  by  the  tests  are  relevant  to  the  job. 

(b)  Test  800  and  801A  may  not  be  used  above  the  GS-3  level. 

(c)  If  the  agency  desires  to  use  the  clerical  ability  test  for  other  positions  or 
levels  prior  approval  of  the  0PM  is  required. 

(3)  Interpretation  of  scores. 

(a)  The  following  information  for  Tests  800  and  801  is  based  on  a  sample  of  applicants 
in  hiring  examinations.   The  mean  was  76.4  and  the  standard  deviation  was  25.3 

(b)  The  table  below  includes  suggested  interpretations  of  total  test  scores. 

Total  Raw  Score  Percent  of  Sample  at  or  , 

(Test  800  &  801A)  above  the  Given  Score  Test  Level 


113 
99 

10 
20 

High 

90 

30 

Good 

82 
75 

40 
50 

Average 

69 

60 

Low 

62  70 

54  80  Poor 

47  90 

Note :   Scores  of  45  or  below  should  be  interpreted  with  caution  since  these 
are  near  or  below  a  chance  level.   It  is  possible  that  such  scores  are  not 
accurate  indicators  of  ability.   Other  factors,  such  as  faulty  test  admin- 
istration or  scoring,  may  have  affected  the  score, 
(c)   The  table  below  includes  suggested  interpretations  of  part  scores  for  use  when 
consideration  of  part  scores  is  relevant  to  the  ranking  process. 

Clerical  Verbal 

Test  800  Test  801A  Test  Level 


52  and  above  52  and  above  High 

44-51  44-51  Average  to  High 

32-43  32-43  Low  to  Average 

31  and  below  31  and  below  Low 
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(d)   The  table  below  can  be  used  for  converting  raw  scores  to  standard  scores. 

Conversion  Table,  Test  800  +  801A 


Raw 
Score 

151 

150 

149 

148 

147 

146 

145 

144 

143 

142 

141 

140 

139 

138 

137 

136 

135 

134 

133 

132 

131 

130 

129 

128 

127 

126 

125 

124 

123 

122 

121 

120 

119 

118 

117 

116 

115 

114 


Standard 
Score 

3.00 
2.96 
2.92 
2.88 
2.84 
2.80 
2.76 
2.72 


.68 
.64 
.60 
,56 
,52 
,48 
,44 
,40 
36 
32 
28 
24 
20 
16 
12 
08 
04 


2.00 
1.96 
1.92 
1.88 
1.84 


80 
76 
72 
68 
64 
60 
56 
52 


Raw 
Score 

113 
112 
111 
110 
109 
108 
107 
106 
105 
104 
103 
102 
101 
100 

99 

98 

97 

96 

95 

94 

93 

92 

91 

90 

89 

88 

87 

86 

85 

84 

83 

82 

81 

80 

79 

78 

77 

76 


Standard 

Raw 

Score 

Score 

1.48 

75 

1.44 

74 

1.40 

73 

1.36 

72 

1.32 

71 

1.28 

70 

1.24 

69 

1.20 

68 

1.16 

67 

1.12 

66 

1.08 

65 

1.04 

64 

1.00 

63 

.96 

62 

.92 

61 

.88 

60 

.84 

59 

.80 

58 

.76 

57 

.72 

56 

.68 

55 

.64 

54 

.60 

53 

.56 

52 

.52 

51 

.48 

50 

.44 

49 

.40 

48 

.36 

47 

.32 

46 

.28 

45 

.24 

44 

.20 

43 

.16 

42 

.12 

41 

.08 

40 

.04 

39 

.00 

38 

Standard 

Raw 

Standard 

Score 

Score 

Score 

-  .04 

37 

-1.56 

-  .08 

36 

-1.60 

-  .12 

35 

-1.64 

-  .16 

34 

-1.68 

-  .20 

33 

-1.72 

-  .24 

32 

-1.76 

-  .28 

31 

-1.80 

-  .32 

30 

-1.84 

-  .36 

29 

-1.88 

-  .40 

28 

-1.92 

-  .44 

27 

-1.96 

-  .48 

26 

-2.00 

-  .52 

25 

-2.04 

-  .56 

24 

-2.08 

-  .60 

23 

-2.12 

-  .64 

22 

-2.16 

-  .68 

21 

-2.20 

-  .72 

20 

-2.24 

-  .76 

19 

-2.28 

-  .80 

18 

-2.32 

-  .84 

17 

-2.36 

-  .88 

16 

-2.40 

-  .92 

15 

-2.44 

-  .96 

14 

-2.48 

-1.00 

13 

-2.52 

-1.04 

12 

-2.56 

-1.08 

11 

-2.60 

-1.12 

10 

-2.64 

-1.16 

9 

-2.68 

-1.20 

8 

-2.72 

-1.24 

7 

-2.76 

-1.28 

6 

-2.80 

-1.32 

5 

-2.84 

-1.36 

4 

-2.88 

-1.40 

3 

-2.92 

-1.44 

2 

-2.96 

-1.48 

1 

-3.00 

-1.52 

0 

-3.04 

(4)   Scoring  Formula 


Test 

800 

801A 


Formula 
R-1/4W 
R  only 


Attachment  to  FPM  Ltr.  335-13   (18) 


c.   Trades  abilities  test. 

(1)  Introduction. 

The  trades  abilities  test  is  designed  to  measure  abilities  required  in  a  wide  vanoty 
of  occupations,  at  levels  which  require  the  performance  of  a  series  of  simple  trades 
and  labor  tasks.   This  test  includes  measures  of  (a)  gross  dexterity,  (b)  alignment 
dexterity,  (c)  shop  arithmetic,  and  (d)  following  directions. 

(2)  Consideration  for  use. 

(a)  Test  100A,  100B,  100D,  and  100E  are  used  for  filling  positions  in  which  the 
ability  to  learn  and  advance  in  the  occupation  is  essential  such  as  helper-trainee 
positions.   Tests  100A  and  100E  are  also  applicable  for  filling  positions  in  which 
the  ability  to  learn  and  advance  is  not  critical.   (See  Job  Qualification  System  for 
Trades  and  Labor  Occupations  Handbook  X-118C) .   A  determination  must  be  made  that 
the  abilities  measured  by  the  test  are  relevant  to  the  job. 

(b)  Only  series  of  Test  100  which  are  numbered  110  or  above  are  approved  for  use. 

(c)  If  the  agency  desires  to  use  the  test  other  than  as  described  above  prior 
approval  of  the  OPM  is  required. 

(3)  Interpretation  scores. 

(a)  The  following  tables  show  how  the  scores  on  the  trades  aptitude  test  may  be 
used  for  assigning  credit  on  individual  job  elements. 

(b)  Table  1  provides  suggested  crediting  on  elements  for  positions  in  which  the 
ability  to  learn  and  advance  in  the  occupation  is  not  essential. 

(c)  Table  2  provides  suggested  crediting  on  elements  for  positions  in  which  the 
ability  to  learn  and  advance  in  the  occupation  is  essential. 

Table  1 

Dexterity  Following  Directions  Ability  Level 

Test  100A  Test  100E 

70  and  above  141  and  above  Superior  (4  points) 


60-69  131-140  Satisfactory  (3  points) 


45-59  111-130  Potential  (2  points) 


35-44  81-110  Weak  (1  point) 
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Shop  Aptitude        Following  Directions 
Tests  100  A,  B,  D,        Test  100E 
and  E 


Dexterity  Tests 
100A  and  B 


Ability  Level 


258  and  above 

141  and  above 

93  and 

above 

Superior  (4  points) 

225-257 

131-140 

73-92 

Satisfactory  (3  point 

187-224 

111-130 

62-72 

Potential  (2  points) 

81-110 

45-61 

Weak  (1  point) 

(4)   Scoring  formula. 


Test 

Formula 

100A 

R-1/2W 

100B 

R-1/4W 

100D 

R  only 

100E 

R-W 
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